<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="../../aosa.css" type="text/css">
    <title>The Performance of Open Source Software: From SocialCalc to EtherCalc</title>
  </head>
  <body>

    <div class="titlebox">
      <h1>The Performance of Open Source Applications<br>From SocialCalc to EtherCalc</h1>
      <p class="author">Audrey Tang</p>
    </div>

    <p><a href="http://ethercalc.net/">EtherCalc</a> is an online spreadsheet system optimized toward simultaneous editing, using SocialCalc as its in-browser spreadsheet engine. Designed by Dan Bricklin (the inventor of spreadsheets), SocialCalc is part of the Socialtext platform, a suite of social collaboration tools for enterprise users.</p>

<p>For the Socialtext team, performance was the primary goal behind SocialCalc's development in 2006. The key observation was this: Client-side computation in JavaScript, while an order of magnitude slower than server-side computation in Perl, was still much faster than the network latency incurred during <span class="caps">AJAX</span> round trips.</p>

<div class="center figure">
<a name="figure-2.1"></a><img src="ethercalc-images/wikicalc-socialcalc.png" alt="Figure 2.1 - WikiCalc and SocialCalc's performance model. Since 2009, advances in JavaScript runtimes have reduced the 50 ms to less than 10 ms." title="Figure 2.1 - WikiCalc and SocialCalc's performance model. Since 2009, advances in JavaScript runtimes have reduced the 50 ms to less than 10 ms." />
</div>

<p class="center figcaption">
<small>Figure 2.1 - WikiCalc and SocialCalc's performance model. Since 2009, advances in JavaScript runtimes have reduced the 50 ms to less than 10 ms.</small>
</p>

<p>SocialCalc performs all of its computations in the browser; it uses the server only for loading and saving spreadsheets. Toward the end of the <a href="http://www.aosabook.org/en/socialcalc.html"><em>Architecture of Open Source Applications</em></a> chapter on SocialCalc, we introduced simultaneous collaboration on spreadsheets, using a simple, chatroom-like architecture.</p>

<div class="center figure">
<a name="figure-2.2"></a><img src="ethercalc-images/multiplayer-socialcalc.png" alt="Figure 2.2 - Multiplayer SocialCalc" title="Figure 2.2 - Multiplayer SocialCalc" />
</div>

<p class="center figcaption">
<small>Figure 2.2 - Multiplayer SocialCalc</small>
</p>

<p>However, as we began to test it for production deployment, we discovered several shortcomings in its performance and scalability characteristics, motivating a series of system-wide rewrites in order to reach acceptable performance. In this chapter, we'll see how we arrived at that architecture, how we used profiling tools, and how we made new tools to overcome performance problems.</p>

<h3 id="design-constraints">Design Constraints</h3>

<p>The Socialtext platform has both behind-the-firewall and on-the-cloud deployment options, imposing unique constraints on EtherCalc's resource and performance requirements.</p>

<p>At the time of this writing, Socialtext requires 2 <span class="caps">CPU</span> cores and 4 <span class="caps">GB</span> <span class="caps">RAM</span> for VMWare vSphere-based intranet deployment. For cloud-based hosting, a typical Amazon <span class="caps">EC2</span> instance provides about twice that capacity, with 4 cores and 7.5 <span class="caps">GB</span> of <span class="caps">RAM</span>.</p>

<p>Behind-the-firewall deployment means that we can't simply throw hardware at the problem in the same way multi-tenant, hosted-only systems did (e.g., DocVerse, which later became part of Google Docs); we can assume only a modest amount of server capacity.</p>

<p>Compared to intranet deployments, cloud-hosted instances offer better capacity and on-demand extension, but network connections from browsers are usually slower and fraught with frequent disconnections and reconnections.</p>

<p>Therefore, the following resource constraints shaped EtherCalc's architecture directions:</p>

<dl>
<dt>Memory</dt>
<dd><p>An event-based server allows us to scale to thousands of concurrent connections with a small amount of <span class="caps">RAM</span>.</p>
</dd>
<dt><span class="caps">CPU</span></dt>
<dd><p>Based on SocialCalc's original design, we offload most computations and all content rendering to client-side JavaScript.</p>
</dd>
<dt>Network</dt>
<dd><p>By sending spreadsheet operations, instead of spreadsheet content, we reduce bandwidth use and allow recovering over unreliable network connections.</p>
</dd>
</dl>

<h2 id="initial-prototype">Initial Prototype</h2>

<p>We started with a WebSocket server implemented in Perl 5, backed by <a href="https://metacpan.org/release/Feersum">Feersum</a>, a <a href="http://software.schmorp.de/pkg/libev.html">libev</a>-based non-blocking web server developed at Socialtext. Feersum is very fast, capable of handling over 10,000 requests per second on a single <span class="caps">CPU</span>. On top of Feersum, we use the <a href="https://metacpan.org/release/PocketIO">PocketIO</a> middleware to leverage the popular Socket.io JavaScript client, which provides backward compatibility for legacy browsers without WebSocket support.</p>

<p>The initial prototype closely resembles a chat server. Each collaborative session is a chatroom; clients sends their locally executed commands and cursor movements to the server, which relays them to all other clients in the same room.</p>

<p>The diagram below shows a typical flow of operation.</p>

<div class="center figure">
<a name="figure-2.3"></a><img src="ethercalc-images/flow-snapshot.png" alt="Figure 2.3 - Prototype server with snapshot mechanism" title="Figure 2.3 - Prototype server with snapshot mechanism" />
</div>

<p class="center figcaption">
<small>Figure 2.3 - Prototype server with snapshot mechanism</small>
</p>

<p>This workaround solved the <span class="caps">CPU</span> issue for new clients, but created a network performance problem of its own, as it taxes each client's upload bandwidth. Over a slow connection, this delays the reception of subsequent commands from the client.</p>

<p>Moreover, the server has no way to validate the consistency of snapshots submitted by clients. Therefore, an erroneous–or malicious–snapshot can corrupt the state for all newcomers, placing them out of sync with existing peers.</p>

<p>An astute reader may note that both problems are caused by the server's inability to execute spreadsheet commands. If the server can update its own state as it receives each command, it would not need to maintain a command backlog at all.</p>

<p>The in-browser SocialCalc engine is written in JavaScript. We considered translating that logic into Perl, but that would have carried the steep cost of maintaining two code bases. We also experimented with embedded <span class="caps">JS</span> engines (<a href="https://metacpan.org/release/JavaScript-V8">V8</a>, <a href="https://metacpan.org/release/JavaScript-SpiderMonkey">SpiderMonkey</a>), but they imposed their own performance penalties when running inside Feersum's event loop.</p>

<p>Finally, by August 2011, we resolved to rewrite the server in Node.js.</p>

<h2 id="porting-to-node.js">Porting to Node.js</h2>

<p>The initial rewrite went smoothly because both Feersum and Node.js are based on the same libev event model, and Pocket.io's <span class="caps">API</span> matches Socket.io closely. It took us only an afternoon to code a functionally equivalent server in just 80 lines of code, thanks to the concise <span class="caps">API</span> offered by <a href="http://zappajs.com/">ZappaJS</a>.</p>

<p>Initial <a href="http://c9s.github.com/BenchmarkTest/">micro-benchmarking</a> showed that porting to Node.js cost us about one half of the maximum throughput. On a typical Intel Core i5 <span class="caps">CPU</span> in 2011, the original Feersum stack handled 5000 requests per second, while Express on Node.js maxed out at 2800 requests per second.</p>

<p>This performance degradation was deemed acceptable for our first JavaScript port, as it wouldn't significantly increase latency for users, and we expect that it will improve over time.</p>

<p>Subsequently, we continued the work to reduce client-side <span class="caps">CPU</span> use and minimize bandwidth use by tracking each session's ongoing state with server-side SocialCalc spreadsheets.</p>

<div class="center figure">
<a name="figure-2.4"></a><img src="ethercalc-images/flow-nodejs.png" alt="Figure 2.4 - Maintaining spreadsheet state with Node.js server" title="Figure 2.4 - Maintaining spreadsheet state with Node.js server" />
</div>

<p class="center figcaption">
<small>Figure 2.4 - Maintaining spreadsheet state with Node.js server</small>
</p>

<h2 id="server-side-socialcalc">Server-side SocialCalc</h2>

<p>The key enabling technology for our work is <a href="https://github.com/tmpvar/jsdom">jsdom</a>, a full implementation of the <span class="caps">W3C</span> document object model, which enables Node.js to load client-side JavaScript libraries within a simulated browser environment.</p>

<p>Using jsdom, it's trivial to create any number of server-side SocialCalc spreadsheets, each taking about 30 <span class="caps">KB</span> of <span class="caps">RAM</span>, running in its own sandbox:</p>

<pre><code>require! &lt;[ vm jsdom ]&gt;
create-spreadsheet = -&gt;
  document = jsdom.jsdom \&lt;html&gt;&lt;body/&gt;&lt;/html&gt;
  sandbox  = vm.createContext window: document.createWindow! &lt;&lt;&lt; {
    setTimeout, clearTimeout, alert: console.log
  }
  vm.runInContext &quot;&quot;&quot;
    #packed-SocialCalc-js-code
    window.ss = new SocialCalc.SpreadsheetControl
  &quot;&quot;&quot; sandbox</code></pre>

<p>Each collaboration session corresponds to a sandboxed SocialCalc controller, executing commands as they arrive. The server then transmits this up-to-date controller state to newly joined clients, removing the need for backlogs altogether.</p>

<p>Satisfied with benchmarking results, we coded a <a href="http://redis.io/">Redis</a>-based persistence layer and launched <a href="http://ethercalc.org/">EtherCalc.org</a> for public beta testing. For the next six months, it scaled remarkably well, performing millions of spreadsheet operations without a single incident.</p>

<p>In April 2012, after delivering a talk on EtherCalc at the <span class="caps">OSDC</span>.tw conference, I was invited by Trend Micro to participate in their hackathon, adapting EtherCalc into a programmable visualization engine for their real-time web traffic monitoring system.</p>

<p>For their use case we created <span class="caps">REST</span> APIs for accessing individual cells with <span class="caps">GET</span>/<span class="caps">PUT</span> as well as POSTing commands directly to a spreadsheet. During the hackathon, the brand-new <span class="caps">REST</span> handler received hundreds of calls per second, updating graphs and formula cell contents on the browser without any hint of slowdown or memory leaks.</p>

<p>However, at the end-of-day demo, as we piped traffic data into EtherCalc and started to type formulas into the in-browser spreadsheet, the server suddenly locked up, freezing all active connections. We restarted the Node.js process, only to find it consuming 100% <span class="caps">CPU</span>, locking up again soon after.</p>

<p>Flabbergasted, we rolled back to a smaller data set, which did work correctly and allowed us to finish the demo. But I wondered: what caused the lock-up in the first place?</p>

<h2 id="profiling-node.js">Profiling Node.js</h2>

<p>To find out where those <span class="caps">CPU</span> cycles went to, we needed a profiler.</p>

<p>Profiling the initial Perl prototype had been very straightforward, thanks largely to the illustrious <a href="https://metacpan.org/module/Devel::NYTProf">NYTProf</a> profiler, which provides per-function, per-line, per-opcode and per-block timing information, with detailed <a href="https://metacpan.org/module/nytprofcg">call-graph visualization</a> and <span class="caps">HTML</span> reports. In addition to NYTProf, we also traced long-running processes with Perl's built-in <a href="https://metacpan.org/module/perldtrace">DTrace support</a>, obtaining real-time statistics on function entry and exit.</p>

<p>In contrast, Node.js's profiling tools leave much to be desired. As of this writing, DTrace support is still limited to <a href="http://blog.nodejs.org/2012/04/25/profiling-node-js/">illumos-based systems</a> in 32-bit mode, so we mostly relied on the <a href="https://github.com/c4milo/node-webkit-agent">Node Webkit Agent</a>, which provides an accessible profiling interface, albeit with only function-level statistics.</p>

<p>A typical profiling session looks like this:</p>

<pre><code># &quot;lsc&quot; is the LiveScript compiler
# Load WebKit agent, then run app.js:
lsc -r webkit-devtools-agent -er ./app.js
# In another terminal tab, launch the profiler:
killall -USR2 node
# Open this URL in a WebKit browser to start profiling:
open http://tinyurl.com/node0-8-agent</code></pre>

<p>To recreate the heavy background load, we performed high-concurrency <span class="caps">REST</span> <span class="caps">API</span> calls with the Apache benchmarking tool <a href="http://httpd.apache.org/docs/trunk/programs/ab.html"><code>ab</code></a>. For simulating browser-side operations, such as moving cursors and updating formulas, we used <a href="http://zombie.labnotes.org/">Zombie.js</a>, a headless browser also built with jsdom and Node.js.</p>

<p>Ironically, the bottleneck turned out to be in jsdom itself.</p>

<!-- : ab -n 10000 /_/1/html # 540rps -->

<div class="center figure">
<a name="figure-2.5"></a><img src="ethercalc-images/profiler-jsdom.png" alt="Figure 2.5 - Profiler screenshot (with jsdom)" title="Figure 2.5 - Profiler screenshot (with jsdom)" />
</div>

<p class="center figcaption">
<small>Figure 2.5 - Profiler screenshot (with jsdom)</small>
</p>

<p>From the report in <a href="#figure-2.5">Figure 2.5</a>, we can see that <code>RenderSheet</code> dominates the <span class="caps">CPU</span> use. Each time the server receives a command, it spends a few microseconds to redraw the <code>innerHTML</code> of cells to reflect the effect of each command.</p>

<p>Because all jsdom code runs in a single thread, subsequent <span class="caps">REST</span> <span class="caps">API</span> calls are blocked until the previous command's rendering completes. Under high concurrency, this queue eventually triggered a latent bug that ultimately resulted in server lock-up.</p>

<p>As we scrutinized the heap usage, we saw that the rendered result is all but unreferenced, as we don't really need a real-time <span class="caps">HTML</span> display on the server side. The only reference to it is in the <span class="caps">HTML</span> export <span class="caps">API</span>, and for that we can always reconstruct each cell's <code>innerHTML</code> rendering from the spreadsheet's in-memory structure.</p>

<p>So, we removed jsdom from the <code>RenderSheet</code> function, re-implemented a minimal <span class="caps">DOM</span> in <a href="https://github.com/audreyt/ethercalc/commit/fc62c0eb#L1R97">20 lines of LiveScript</a> for <span class="caps">HTML</span> export, then ran the profiler again (see <a href="#figure-2.6">Figure 2.6</a>).</p>

<!-- ab -n 10000 /_/1/html # 2116rps -->

<div class="center figure">
<a name="figure-2.6"></a><img src="ethercalc-images/profiler-no-jsdom.png" alt="Figure 2.6 - Updated profiler screenshot (without jsdom)" title="Figure 2.6 - Updated profiler screenshot (without jsdom)" />
</div>

<p class="center figcaption">
<small>Figure 2.6 - Updated profiler screenshot (without jsdom)</small>
</p>

<p>Much better! We have improved throughput by a factor of 4, <span class="caps">HTML</span> exporting is 20 times faster, and the lock-up problem is gone.</p>

<h2 id="multi-core-scaling">Multi-core Scaling</h2>

<p>After this round of improvement, we finally felt comfortable enough to integrate EtherCalc into the Socialtext platform, providing simultaneous editing for wiki pages and spreadsheets alike.</p>

<p>To ensure a fair response time on production environments, we deployed a reverse-proxying nginx server, using its <a href="http://wiki.nginx.org/HttpLimitReqModule#limit_req"><code>limit_req</code></a> directive to put an upper limit on the rate of <span class="caps">API</span> calls. This technique proved satisfactory for both behind-the-firewall and dedicated-instance hosting scenarios.</p>

<p>For small and medium-sized enterprise customers, though, Socialtext provides a third deployment option: multi-tenant hosting. A single, large server hosts more than 35,000 companies, each averaging around 100 users.</p>

<p>In this multi-tenant scenario, the rate limit is shared by all customers making <span class="caps">REST</span> <span class="caps">API</span> calls. This makes each client's effective limit much more constraining–around 5 requests per second. As noted in the previous section, this limitation is caused by Node.js using only one <span class="caps">CPU</span> for all its computation.</p>

<div class="center figure">
<a name="figure-2.7"></a><img src="ethercalc-images/scaling-evented.png" alt="Figure 2.7 - Event server (single-core)" title="Figure 2.7 - Event server (single-core)" />
</div>

<p class="center figcaption">
<small>Figure 2.7 - Event server (single-core)</small>
</p>

<p>Is there a way to make use of all those spare CPUs in the multi-tenant server?</p>

<p>For other Node.js services running on multi-core hosts, we utilized a pre-forking <a href="https://npmjs.org/package/cluster-server">cluster server</a> that creates a process for each <span class="caps">CPU</span>.</p>

<div class="center figure">
<a name="figure-2.8"></a><img src="ethercalc-images/scaling-cluster.png" alt="Figure 2.8 - Event cluster server (multi-core)" title="Figure 2.8 - Event cluster server (multi-core)" />
</div>

<p class="center figcaption">
<small>Figure 2.8 - Event cluster server (multi-core)</small>
</p>

<p>However, while EtherCalc does support multi-server scaling with Redis, the interplay of <a href="http://stackoverflow.com/a/5749667">Socket.io clustering</a> with <a href="https://github.com/LearnBoost/Socket.IO/wiki/Configuring-Socket.IO">RedisStore</a> in a single server would have massively complicated the logic, making debugging much more difficult.</p>

<p>Moreover, if all processes in the cluster are tied in <span class="caps">CPU</span>-bound processing, subsequent connections would still get blocked.</p>

<p>Instead of pre-forking a fixed number of processes, we sought a way to create one background thread for each server-side spreadsheet, thereby distributing the work of command execution among all <span class="caps">CPU</span> cores.</p>

<div class="center figure">
<a name="figure-2.9"></a><img src="ethercalc-images/scaling-threads.png" alt="Figure 2.9 - Event threaded server (multi-core)" title="Figure 2.9 - Event threaded server (multi-core)" />
</div>

<p class="center figcaption">
<small>Figure 2.9 - Event threaded server (multi-core)</small>
</p>

<p>For our purpose, the <span class="caps">W3C</span> <a href="http://www.w3.org/TR/workers/">Web Worker</a> <span class="caps">API</span> is a perfect match. Originally intended for browsers, it defines a way to run scripts in the background independently. This allows long tasks to be executed continuously while keeping the main thread responsive.</p>

<p>So we created <a href="https://github.com/audreyt/node-webworker-threads">webworker-threads</a>, a cross-platform implementation of the Web Worker <span class="caps">API</span> for Node.js.</p>

<p>Using webworker-threads, it's very straightforward to create a new SocialCalc thread and communicate with it:</p>

<pre><code>{ Worker } = require \webworker-threads
w = new Worker \packed-SocialCalc.js
w.onmessage = (event) -&gt; ...
w.postMessage command</code></pre>

<p>This solution offers the best of both worlds: It gives us the freedom to allocate more CPUs to EtherCalc whenever needed, and the overhead of background thread creation remains negligible on single-<span class="caps">CPU</span> environments.</p>

<h2 id="lessons-learned">Lessons Learned</h2>

<h3 id="constraints-are-liberating">Constraints are Liberating</h3>

<p>In his book <em>The Design of Design</em>, Fred Brooks argues that by shrinking the designer's search space, constraints can help to focus and expedite a design process. This includes self-imposed constraints:</p>

<blockquote>
<p>Artificial constraints for one's design task have the nice property that one is free to relax them. Ideally, they push one into an unexplored corner of a design space, stimulating creativity.</p>
</blockquote>

<p>During EtherCalc's development, such constraints were essential to maintain its <em>conceptual integrity</em> throughout various iterations.</p>

<p>For example, it might seem attractive to adopt three different concurrency architectures, each tailored to one of our server flavors (behind-the-firewall, on-the-cloud, and multi-tenant hosting). However, such premature optimization would have severely hampered the conceptual integrity.</p>

<p>Instead, I kept the focus on getting EtherCalc performing well without trading off one resource requirement for another, thereby minimizing its <span class="caps">CPU</span>, <span class="caps">RAM</span> and network uses at the same time. Indeed, since the <span class="caps">RAM</span> requirement is under 100 <span class="caps">MB</span>, even embedded platforms such as Raspberry Pi can host it easily.</p>

<p>This self-imposed constraint made it possible to deploy EtherCalc on PaaS environments (e.g., DotCloud, Nodejitsu and Heroku) where all three resources are constrained instead of just one. This made it very easy for people to set up a personal spreadsheet service, thus prompting more contributions from independent integrators.</p>

<h3 id="worst-is-best">Worst is Best</h3>

<p>At the <span class="caps">YAPC</span>::<span class="caps">NA</span> 2006 conference in Chicago, I was invited to predict the open-source landscape, and this was my <a href="http://pugs.blogs.com/pugs/2006/06/my_yapcna_light.html">entry</a></p>

<blockquote>
<p>I think, but I cannot prove, that next year JavaScript 2.0 will bootstrap itself, complete self-hosting, compile back to JavaScript 1, and replace Ruby as the Next Big Thing on all environments.</p>
<p>I think <span class="caps">CPAN</span> and <span class="caps">JSAN</span> will merge; JavaScript will become the common backend for all dynamic languages, so you can write Perl to run in the browser, on the server, and inside databases, all with the same set of development tools.</p>
<p>Because, as we know, <em>worse is better</em>, so the <em>worst</em> scripting language is doomed to become the <em>best</em>.</p>
</blockquote>

<p>The vision turned into reality around 2009 with the advent of new JavaScript engines running at the speed of native machine instructions. By the time of this writing, JavaScript has become a <em>"write once, run anywhere"</em> virtual machine–other major languages can compile to it with <a href="http://asmjs.org">almost no performance penalty</a>.</p>

<p>In addition to browsers on the client side and Node.js on the server, JavaScript also <a href="http://pgre.st/">made headway</a> into the Postgres database, enjoying a large collection of freely reusable <a href="https://npmjs.org/">modules</a> shared by these runtime environments.</p>

<p>What enabled such sudden growth in the community? During the course of EtherCalc's development, from participating in the fledgling <span class="caps">NPM</span> community, I reckoned that it was precisely because JavaScript prescribes very little and bends itself to the various uses, so innovators can focus on the vocabulary and idioms (e.g., jQuery and Node.js), each team abstracting their own <em>Good Parts</em> from a common, liberal core.</p>

<p>New users are offered a very simple subset to begin with; experienced developers are presented with the challenge to evolve better conventions from existing ones. Instead of relying on a core team of designers to get a complete language right for all anticipated uses, the grassroots development of JavaScript echoes Richard P. Gabriel's well-known maxim of <a href="http://www.dreamsongs.com/WorseIsBetter.html">Worse is Better</a>.</p>

<h3 id="livescript-redux">LiveScript, Redux</h3>

<p>In contrast to the straightforward Perl syntax of <a href="https://metacpan.org/module/Coro::AnyEvent">Coro::AnyEvent</a>, the callback-based <span class="caps">API</span> of Node.js necessitates deeply nested functions that are difficult to reuse.</p>

<p>After experimenting with various flow-control libraries, I finally solved this issue by settling on <a href="http://livescript.net/">LiveScript</a>, a new language that compiles to JavaScript, with syntax heavily inspired by Haskell and Perl.</p>

<p>In fact, EtherCalc was ported through a lineage of four languages: JavaScript, CoffeeScript, Coco and LiveScript. Each iteration brings more expressivity, while maintaining full back and forward compatibility, thanks to efforts such as <a href="http://js2coffee.org/">js2coffee</a> and <a href="http://js2ls.org/">js2ls</a>.</p>

<p>Because LiveScript compiles to JavaScript rather than interpreting its own bytecode, it remains completely compatible with function-scoped profilers. Its generated code performs as good as hand-tuned JavaScript, taking full advantage of modern native runtimes.</p>

<p>On the syntactic side, LiveScript eliminated nested callbacks with novel constructs such as <a href="http://livescript.net/#backcalls">backcalls</a> and <a href="http://livescript.net/#cascades">cascades</a>. It provides us with powerful syntactic tools for functional and object-oriented composition.</p>

<p>When I first encountered LiveScript, I remarked that it's like "a smaller language within Perl 6, struggling to get out"–a goal made much easier by adopting the same semantics as JavaScript itself and focusing strictly on syntactical ergonomics.</p>

<h3 id="conclusion">Conclusion</h3>

<p>Unlike the SocialCalc project's well-defined specification and team development process, EtherCalc was a solo experiment from mid-2011 to late-2012 and served as a proving ground for assessing Node.js's readiness for production use.</p>

<p>This unconstrained freedom afforded an exciting opportunity to explore a wide variety of alternative languages, libraries, algorithms and architectures. I'm very grateful to all contributors, collaborators and integrators, and especially to Dan Bricklin and my Socialtext colleagues for encouraging me to experiment with these technologies. Thank you, folks!</p>

  </body>
</html>
